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EXECUTIVE  SUMMARY 


A.  BACKGROUND.  Selecting  a  strategy  for  the  production  phase  of  a  weapon 
system  acquisition  is  one  of  the  most  important  decisions  in  successfully 
achieving  a  system's  fielding  objectives.  However,  the  decision  and  its 
outcomes  are  nonobvious,  and  existing  analytical  models  used  to  assist  in 
the  decision-making  process  do  not  provide  for  the  systematic  analysis  of 
each  available  option.  Neither  do  the  models  consider  all  three  parts  of 
the  decision  environment:  potential  strategies,  objectives,  and  system 
specific  conditions.  What  the  decision  analysis  demands  is  an  approach  that 
realistically  deals  with  both  the  complexities  and  uncertainties  present. 

B.  STUDY  OBJECTIVES.  The  objectives  of  this  study  were  to:  (1)  define  the 
production  acquisition  strategy  problem;  (2)  develop  a  structured  approach 
to  the  decision  process;  (3)  develop  a  mathematical  reDresentation  of  the 
cost  outcomes;  (4)  report  findings  on  strategy  impacts  on  non-cost  issues; 
(5)  develop  a  screening  methodology  to  focus  the  decision  process  on  key 
alternatives;  and  (6)  discuss  the  appl icabil ity  of  Venture  Evaluation  and 
Review  Technique  (VERT)  in  the  analysis  effort. 

C.  REPORT  METHODOLOGY.  Research  began  with  a  literature  review  of  previous 
studies  on  strategy  decisions,  competition  cost  savings,  and  uncertainty 
analysis.  Limitations  in  the  existing  studies  were  identified  to  determine 
what  aspects  of  the  decision  process  required  more  complete  treatment.  A 
requirement  for  empirical  data  to  analyze  non-cost  factors  was  established 
from  this  effort.  A  cost  model  incorporating  cost  improvement  curves  and 
uncertainty  analysis  was  created  for  use  in  analyzing  the  alternative 
strategies.  A  VERT  analysis  was  included  to  demonstrate  the  advantages  of 
using  a  coherent  framework  in  a  complex  analysis  effort. 

D.  CONCLUSIONS  AND  RECOMMENDATIONS.  The  empirical  research  on  non-cost 
issues,  specifically  reliability  and  schedule,  was  limited  by  the  few 
qualifying  systems  with  auditable  records.  Tentative  conclusions  were  that 
(1)  given  government  monitoring  the  reliability  objective  should 
typically  be  given  little  weight  in  the  strategy  decision  and  (2)  second 
sources  do  experience  initial  schedule  delays  of  greater  magnitudes  than 
the  developer.  The  study  presents  an  approach  to  analyzing  the  strategy 
selection  problem,  A  screening  technique  is  used  to  focus  the  detailed 
analysis  on  the  viable  strategies,  The  cost  model  integrates  inputs  from 
the  areas  of  methodology,  historical  data,  and  judgment.  It  is  shown  that 

a  stochastic  network  can  be  used  to  represent  the  dynamics  of  the  acquisition 
process.  The  study  recommends  the  cost-saying  behavior  of  multi-year 
contracts  be  investigated. 
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INTRODUCTION 


A.  PROBLEM  DESCRIPTION. 

Selecting  an  acquisition  strategy  for  a  weapon  system  is  one  of  the 
most  important  decisions  in  making  the  fielding  of  that  system  a  reality. 

The  acquisition  strategy  decision  has  far  ranging  impacts  on  the  possibili¬ 
ties  of  generating  competitive  cost  savings  [10]  and  the  degree  of  success 
in  meeting  performance  and  schedule  parameters.  Since  the  decision  and  its 
outcomes  are  nonobvious,  the  decision  should  be  made  only  after  a  systematic 
analysis  of  each  available  option. 

Existing  analytical  models  used  to  assist  in  the  decision  making  process 
do  not  fulfill  this  requirement.  They  generally  accomplish  only  part  of  the 
necessary  analysis  (e.g.,  production  cost)  or  in  some  cases  fail  to  correctly 
interpret  the  data.  None  consider  all  three  parts  of  the  decision  environ¬ 
ment:  (1)  the  potential  strategies,  (2)  the  various  objectives  they  would 
affect  or  achieve,  and  (3)  the  special  conditions  describing  a  specific 
system.  The  purpose  of  this  report  is  to  present  an  approach  to  this 
decision  problem  that  realistically  deals  with  both  the  complexities  and 
the  uncertainties  present  at  the  time  a  decision  must  be  made. 

B.  BACKGROUND/SCOPE. 

To  better  understand  the  complexities  involved  in  the  decision 
environment,  the  three  parts  are  discussed  below. 

1 .  Acquisition  Strategy. 

In  Cox  and  Hullander  [4],  an  acquisition  strategy  was  a  vector 
quantity  (Tq,  Tp  T2,  T3)  representing  the  selection  of  a  particular 
strategy  for  each  of  the  four  major  phases  of  a  system  acquisition: 


concept  development,  demonstration  and  validation,  full-scale  development, 
and  production  and  deployment.  The  strategies  suggested  by  Cox  and 
Hullander  are  displayed  in  Figure  1.  Interdependencies  between  the  T^'s 
may  exist  in  the  sense  that  a  decision  for  an  early  phase  may  preclude  the 
use  of  a  particular  strategy  in  a  later  phase. 

The  scope  of  this  study  is  limited  to  the  selection  of  a  strategy  for 
the  production  phase  (T3)  of  system  acquisition.  Major  portions  of  the 
total  cost  are  encountered  in  the  production  phase.  Earlier  studies 
(e.g.,  10)  collect  and  analyze  data  pertaining  to  this  phase.  This  study 
attempts  to  provide  a  more  complete  perspective  on  the  production  strategy 
problem. 

The  set  of  production  strategies  in  Figure  1  has  been  expanded  as 
shown  in  Figure  2.  Further,  the  concept  of  a  production  strategy  is 
expanded.  Production  may  be  accomplished  through  a  series  of  buys.  Thus, 
a  production  strategy  itself  can  be  regarded  as  a  vector  quantity.  Addi¬ 
tionally,  at  the  level  of  the  contracting  officer,  tactics  must  be  chosen 
to  support  the  strategic  choice  [23]. 

2.  Acquisition  Objectives. 

The  objectives  of  the  acquisition  comprise  the  second  major  part  of 
the  environment.  A  hierarchy  of  general  objectives  relevant  to  the 
production  strategy  decision  is  shown  schematically  in  Figure  3.  Each 
strategy  will  have  different  effects  according  to  the  objective  it  is 
measured  against  as  the  following  examples  show.  Existing  data  has  shown 
that  recurring  production  costs  tend  to  be  reduced  by  competitive 
strategies  [5,  7,  10].  The  time  at  which  the  system  is  fielded  can  also 
be  affected.  In  some  cases  this  timing  impact  may  best  be  measured  as  the 
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Tg/Phase  0:  CONCEPT  EXPLORATION 

•  Directed  Concept 

•  By  Non- Industrial  Firms 

•  By  Industrial  Firms 

T^/ PHASE  1:  DEMONSTRATION  AND  VALIDATION  (D&V) 

•  Waive  D&V 

•  Contract  Definition 

By  Non- Industrial  Firm 
Industrial  Firm  (Single  Source) 
Industrial  Firms  (Multiple  Sources) 

§  Subsystem  and/or  Component  Development 

By  Non-Industrial  Firm 
Industrial  Firm  (Single  Source) 
Industrial  Firms  (Multiple  Sources) 

•  System  Prototypes 

By  Non-Industrial  Firm 
By  Industrial  Firm  (Single  Source) 

By  Industrial  Firms  (Multiple  Sources) 

To/PHASE  2:  FULL-SCALE  DEVELOPMENT  (FSD) 

•  Incremental  Development 

Single  Source 
Multiple  Sources 

•  Concurrent  Development 

Single  Source 
Multiple  Sources 

T  /PHASE  3:  PRODUCTION  &  DEPLOYMENT 

•  Single  Source,  No  Options 

•  Single  Source  with  Options 

•  Single  Source,  Multi-Year  Contract 

•  Leader-Follower 

•  Licensing 

•  Second  Sourcing 


Figure  1 . 


Acquisition  Strategy  Alternatives 


SOURCE:  Cox  and  Hul lander 
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Sole  Source 


Single  Source,  No  Options 
Single  Source  with  options 
Single  Source,  multi-year  contract 


Multiple  Sources 

Leader- Follower 
Licensing 

Technical  Data  Package  (TUP) 
Form,  fit,  function  (F3) 
Teaming 


Figure  2.  General  Production  Strategies. 
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Figure  3.  Hierarchy  of  Acquisition  Objectives. 
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date  by  which  a  "critical  mass"  of  the  system  is  fielded.  Likewise,  the 
strategy  decision  can  affect  the  ultimate  operational  reliability  of  the 
system.  For  example,  a  second  source  may  produce  a  less  reliable  product 
due  to  a  deficiency  in  its  manufacturing  process  below  the  level  of  detail 
specified  by  the  technical  data  package.  Equally,  in  any  given  setting 
specific  objectives  may  be  paramount.  For  example,  it  may  be  critical  that 
a  multi-source  mobilization  capability  exist.  The  objectives  vary  in 
importance  from  case  to  case.  Cost,  however,  is  almost  always  considered 
important,  and  it  is  the  objective  that  most  fully  needs  and  is  suited  to 
a  comprehensive  modeling  effort. 

3.  System  Conditions. 

In  addition  to  strategies  and  objectives,  the  third  major  defining 
topic  is  the  conditions  present  for  the  system.  A  suggested  list  of  condi¬ 
tions  that  affect  the  various  outcome  objectives  is  provided  in  Figure  4. 
Other  authors  [4,  16,  17,  23]  have  also  suggested  critical  conditions  in  a 
system  acquisition. 

4 .  Key  Factors  of  Strategy  Selection  Studies. 

Earlier  studies  have  looked  at  the  strategy  selection  problem  and 
especially  at  the  relative  costs  of  competitive  and  non-competitive 
production  strategies.  The  proposed  decision  models,  however,  have  been 
too  simplistic  in  their  analysis  of  the  problem.  A  model  for  the  strategy 
selection  decision  should  recognize  several  key  factors  associated  with  the 
problem. 

One  of  the  factors  of  major  significance  in  system  cost  estimating 
is  the  phenomenon  of  learning,  or  cost  improvement,  over  time.  Cost 
improvement  resulting  from  production  experience  must  play  a  central  role 
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1.  Affecting  Life  Cycle  Cost: 


Complexity 

Quantity 

Duration 

Non-recurring  costs 
to  establish  a  source 
Extent  of  subcontracting 
Nature  of  Production  Process 
Extent  of  FMS 
Technical  data  rights 

2.  Affecting  Performance: 
Complexity 

Extent  of  subcontracting 
Nature  of  production  process 
Commodity  Class 
Technical  maturity  at  beg. 

of  production 
Quality  of  TDP 

3.  Affecting  IOC: 

Complexity 
Commodity  class 
Technical  maturity  at  beg. 
of  production 

4.  Affecting  mobilization  goal: 
Capacity 

5.  Broad  goals: 

NATO  agreements 

State  of  industrial  base 

State  of  general  economy 


Funding 

Commodity  class 
Technical  maturity  at  beg.  of 
production 
Capacity 

Learning  curve  sloDes 
Quality  of  TDP 
No.  of  qualified  Krs 
Attitude  of  Prime 


No.  of  qual ified  Krs. 

Attitude  of  prime 

Cost  to  develop  2nd  source 

Time  to  develop  2nd  source 

Kr's  past  performance 

Commonality 


Quality  of  TDP 

No.  of  qualified  Krs. 

Attitude  of  prime 


Figure  4.  Significant  System  Conditions. 
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in  any  cost-oriented  decision  model.  Usually  analysts  have  assumed  a  power 
function  cost  improvement  curve  (y  -  ax^).  Such  a  curve  results  from  the 
assumption  that  every  doubling  of  cumulative  production  quantity  results  in 
a  fixed  percentage  reduction  in  unit  cost.  In  recent  years  all  serious  cost 
efforts  in  defense  systems  acquisition  have  incorporated  the  cost  improve¬ 
ment  curve  notion. 

A  second  factor--production  rate  effects  on  system  cost--has  not 
achieved  the  same  recognition  as  the  cost  improvement  curve.  Yet  it  is 
important  to  include  this  factor  in  comparing  alternatives  involving  sig¬ 
nificantly  different  production  rates.  For  examole,  dual  sourcing 
strategies  result  in  lower  rates  for  individual  oroducers.  Several 
different  approaches  have  been  suggested  for  1  inkina  the  rate  effect  with 
the  learning  effect  [1,  2,  18].  'Thile  the  Droposed  treatments  have  differed, 
actual  cases  have  demonstrated  that  in  the  current  defense  environment 
production  stretchouts  with  their  lower  annual  quantities  lead  to  higher  costs. 

A  third  key  factor  in  the  problem  is  the  form  taken  by  the  competi¬ 
tive  buy.  It  is  clear  that  greater  competitive  pressures  exist  in  some 
contracting  situations  than  in  others.  Further,  the  extent  to  which 
competitive  prices  are  reduced  below  sole  source  levels  ought  to  reflect 
the  necessarily  subjective  degree  of  competitive  pressure. 

The  three  factors  discussed  above--learning,  production  rate,  and 
competitive  envi ronment--are  key  factors  affecting  the  outcomes  of  strategy 
selection  decisions.  It  is  important  to  dissect  the  problem  and  understand 
how  these  factors  manifest  themselves  even  if  objective  data  do  not  provide 
satisfactory  estimates  of  their  values  in  a  given  situation. 

Several  difficulties  impinge  upon  the  ability  to  formulate  a 
decision  model  providing  insight  into  the  effect  of  alternate  production 
strategies.  For  example,  not  nearly  enough  programs  have  been  executed 
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and  documented  under  different  production  strategies  to  provide  statistical 
data  on  individual  strategies  under  particular  sets  of  conditions.  For 
obvious  economic  reasons,  it  is  infeasible  to  obtain  data  via  experiments. 

Therefore,  an  approach  must  be  devised  that  enables  one  to  extract  the  very 
maximum  information  available.  The  model  proposed  in  this  report  can  help 
in  analyzing  the  relative  effects  of  the  primary  components  of  the  problem. 

It  is  an  integrated  model  in  the  sense  that  model  inputs  depend  on  a  blend 
of  judgment  and  existing  quantitative  data. 

5.  Assessment  of  Historical  Work, 

Several  areas  of  research  effort  have  provided  a  stimulus  for 
the  present  work.  Key  documents  are  described  in  the  following  paragraphs, 
a.  Competition  Data  and  Cost  Models. 

A  fundamental  study  concerned  with  estimating  the  savings 
resulting  from  production  competition  in  defense  system  acquisition  was 
Lovett  and  Norton  [10].  Prior  to  this  effort,  assessments  of  competitive 
savings  had  typically  been  done  in  crude  ways  that  did  not  reflect  factors 
such  as  sole-source  learning.  Recently  other  studies  have  analyzed  the 
empirical  findings  from  competitive  system  procurements  [5,  7].  The 
different  studies  agree  in  their  findings  of  observed  savings  and  the  high 
degree  of  variability  in  these  sayings,  These  studies  form  the  primary 
empirical  base  for  estimates  of  recurring  cost  savings  behavior,  Previous 
attempts  to  go  beyond  the  empirical  sayings  to  predictive  models,  however, 
have  shortcomings.  This  topic  will  be  discussed  in  Chapter  HI  when  a 
modeling  approach  is  proposed, 

The  most  complete  case  analysis  of  the  second  sourcing  decision  for 
production  seems  to  be  the  MLRS  Second  Sourcing  Rocket  Acquisition  Study  [121. 
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The  model  proposed  in  this  report  attempts  to  extend  [12]  by  providing  a 
more  comprehensive  framework  for  the  uncertainty  present  in  costino. 

The  analysis  of  non-cost  issues  in  [12]  can  also  serve  as  useful  back¬ 
ground  for  analysts  doing  a  strategy  selection  study. 

b.  Acquisition  Conditions. 

The  contingency  theory  of  management  is  explored  by  Hofer  [81 
and  Luthans  [11]  among  others.  This  approach  attempts  to  relate  the 
appropriate  management  strategy  to  the  existing  conditions.  Williams  and 
Knittle  [23]  suggest  this  approach  for  defense  systems,  but  their  results 
remain  too  general  for  operational  use.  Parry  and  Sellers  [16,  171  focus 
more  specifically  on  the  conditions  favorable  or  unfavorable  to  second 
sourcing,  but  they  do  not  deal  with  the  relative  importance  and  magnitude 
of  the  various  conditions,  Sparks  [21]  proposed  a  condition-oriented  model 
for  licensing,  but  the  model  leaves  some  of  the  fundamental  issues  to  be 
answered  with  simple  yes-no  responses.  A  few  other  such  models  have  been 
proposed[14,  22]. 

c.  Subsidiary  Problems. 

The  relationship  of  production  rate  to  the  strategy  selection 
problem  was  discussed  earlier.  Other  areas  which  may  bear  on  a  given 
strategy  decision  are  described  below,  One  of  these  is  the  problem  of 
allocating  production  quantities  among  sources  when  attempting  to 
sustain  both  a  competitive  environment  and  dual  sources.  Sol  insky  discussed 
this  problem  and  recommended  an  allocation  rule  in  [20],  A  second  sub¬ 
problem  deals  with  price  improvement  curve  slopes  under  competitive 
strategies  relative  to  sole  source  strategies.  In  [19]  Smith  and  Lowe 
concluded  that  sole  source  and  competitive  slopes  should  ordinarily  be 
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treated  as  equal  or  with  slight  improvement  for  competition. 

d.  Decision  Modeling. 

Some  studies  from  decision-making  literature  are  also  relevant 
to  the  current  research.  Park  et  al.  [15]  suggested  a  decision  plan  net 
that  served  as  a  guide  to  the  decision  screen  part  of  the  approach 
discussed  in  Chapter  III.  The  screen,  as  applied  in  this  report,  reduces 
the  strategies  to  a  smaller  set  capable  of  more  detailed  evaluation.  The 
primary  emphasis  in  this  report  is  on  structuring  an  appropriate  model 
regardless  of  the  relative  weight  given  differing  objectives  in  a  particular 
case.  DeWispelare  and  Sage  [6]  suggest  a  multi-attribute  utility  approach 
to  a  different  defense  acquisition  problem.  Much  of  their  work  could  be 
applied  to  preference  evaluation  when  preferences  cannot  be  easily  assessed 
directly  from  predicted  outcomes.  Thus,  their  approach  could  be  used  in 
conjunction  with  the  effort  described  in  this  report.  The  Venture  Evalua¬ 
tion  and  Review  Technique  (VERT)  is  used  to  analyze  the  probabilistic 
network  describing  an  acquisition  strategy  in  Chapter  III.  This  technique 
was  developed  by  Moeller  [13]. 

e.  Assessment  Summary. 

In  summary,  previously  proposed  models  for  system  strategy 
selection  have  lacked  sufficient  content  to  be  a  useful  operational  guide 
to  project  management.  The  work  has  been  heavily  process-oriented  and 
non-quantitative.  This  effort  tries  to  bring  the  problem  closer  to  an 
operational  solution  through  explicit  analysis  coupling  systematically 
judgment  and  empirical  observations.  A  real  dearth  of  work  exists  treating 
multi-year  procurement  which  is  likely  to  see  increased  use  in  the 
future  [3]. 
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C.  STUDY  OBJECTIVES. 


The  objectives  of  this  study  were  to: 

1.  Define  the  production  acquisition  strategy  problem  to  include 
identifying  the  interrelationships  between  strategies,  system  conditions 
and  objectives. 

2.  Discuss  previous  approaches  to  the  strategy  decision  and  their 
limitations. 

3.  Report  findings  on  strategy  impact  on  non-cost  issues. 

4.  Develop  a  structured  approach  to  the  decision  process  which 
recognizes  the  dynamics  of  the  factors  and  the  uncertainty  involved. 

5.  Develop  a  mathematical  representation  of  the  cost  outcomes. 

6.  Develop  a  production  acquisition  strategy  screening  methodology  to 
focus  the  decision  process  on  key  alternatives. 

7.  Discuss  the  applicability  of  various  analytical  tools  with  emphasis 
on  the  Venture  Evaluation  and  Review  Technique  (VERT). 
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CHAPTER  II 


RELIABILITY  AND  SCHEDULE  FINDINGS 
ABOUT  SECOND  PRODUCTION  SOURCES 


A.  GENERAL. 

Earlier  empirical  research  related  to  acquisition  strategy  decision¬ 
making  has  focuv^d  on  the  cost  dimension.  Nothing  systematic  was  found 
that  focused  on  the  attainment  of  non-cost  objectives.  In  this  chapter 
data  bearing  on  the  other  major  dimensions  of  the  decision--schedule  and 
reliability--are  examined.  Cases  providing  schedule  and  reliability 
information  about  second  source  production  proved  to  be  extremely  limited. 
The  cases  for  which  data  proved  obtainable  were  projects  with  the  Naval 
Material  Command  and  the  US  Army  Materiel  Development  and  Readiness  Command. 
The  findings  are  insufficient  for  rigorous  analysis,  but  the  charts  and 
graphs  recording  the  actual  observations  are  presented  in  Appendices  A  and 

B.  The  identity  of  the  contractors  involved  is  not  revealed.  Attention 
should  be  focused  on  the  phenomena  observed  and  not  on  which  party  may  have 
been  involved.  All  the  cases  were  second  sourcing  of  the  TDP  (technical 
data  package)  type  in  which  an  exact  copy  was  made  by  the  second  source. 

B.  RELIABILITY. 

The  second  sources  sometimes  encountered  reliability  problems  early  in 
production.  These  were  generally  corrected  with  little  or  no  degradation 
of  operational  reliability.  It  must  be  noted  that  these  cases  reflect 
situations  where  competition  was  actually  undertaken.  Therefore,  some 
basic  conditions  must  have  been  previously  met.  For  example,  a  determina¬ 
tion  would  have  been  made  that  the  source  was  technically  qualified.  Note 
too  that  the  aspect  of  reliability  bearing  on  the  strategy  decision  is  the 
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operational  reliability  of  the  second  source's  product  relative  to  the  first 
source.  Under  some  options  it  may  be  that  the  second  source  is  producing 
a  different  product  than  the  original  developer.  However,  as  mentioned 
above,  all  of  the  data  presented  here  deals  with  identical  TDP  copies. 

An  important  point  is  that  deficiencies  in  the  technical  data  packages 
were  commonly  acknowledged  by  Government  personnel.  Contracting-oriented 
persons  seemed  frequently,  however,  to  have  a  narrow  and  mistaken  view  that 
a  quality  technical  data  package  and  a  well -prepared  contract  would  resolve 
all  the  problems.  It  is  apparent,  though,  that  even  a  quality  technical 
data  package  is  not  sufficient  to  insure  the  absence  of  production  problems. 
At  least  two  reasons  have  been  offered  to  explain  this.  In  the  first  place, 
continual  corrections  are  made  to  the  TDP  in  the  early  stages  of  production. 
Some  of  these  corrections  may  fail  to  be  fed  to  the  second  source.  Secondly, 
the  level  of  detail  of  a  TDP  is  not  sufficient  to  explain  all  production 
processes. 

Appendix  A  documents  the  situations  the  data-gathering  effort  uncovered. 
The  tentative  conclusion  is  that  the  weight  of  reliability  considerations 
in  the  strategy  selection  decision  should  typically  be  very  small  provided 
the  government  monitors  reliability.  Most  problems  are  uncovered  in 
system  acceptance  testing  and  do  not  reach  the  field  to  affect  capability. 

C.  SCHEDULE. 

The  date  at  which  a  "critical  mass"  of  a  system  is  fielded  may  be  used 
to  measure  strategy  differences  in  the  schedule  area.  The  definition  of  a 
critical  mass  may  vary  with  the  system,  but  it  typically  might  involve  the 
fielding  of  many  more  product  units  than  the  initial  operating  capability 
(IOC)  entails.  IOC  is  a  commonly  used  date  for  reporting  progress  on 
major  systems. 
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There  are  two  ways  in  which  delivery  time  of  the  system  can  be  affected 
by  a  second  source  strategy.  First,  an  alternative  production  strategy  may 
require  a  new  fielding  schedule.  For  example,  additional  time  may  be 
required  for  adequately  establishing  a  second  source.  The  other  effect  is 
on  the  risk  of  meeting  planned  delivery  schedules.  These  two  effects  are 
illustrated  in  Figure  5. 


A.  PRODUCTION  SCHEDULE  CHANGE  (SHIFT  OF  MEAN) 

B.  RISK  CHANGE  (INCREASED  VARIANCE) 

Figure  5.  Example  Schedule  Effect. 

Commonly,  the  second  sources  encountered  an  initial  period  of  difficulty 
in  meeting  their  production  delivery  schedule.  The  graphs  in  Appendix  B 
suggest  the  extent  and  duration  of  this  problem.  For  the  six  systems 
graphed  in  Appendix  B  the  second  production  sources  were  consistently  more 
delinquent  than  the  original  source  at  comparable  points  in  their  produc¬ 
tion  experience.  The  results  varied  widely,  but  the  difference  between 
the  sources  in  deliveries  versus  schedule  was  typically  three  to  five 
months.  Informal  comments  are  also  provided  in  Appendix  B  applicable  to 
some  of  the  systems. 
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The  suggested  conclusion  from  the  data  is  that  the  technical  data 
package  does  not  solve  all  production-related  issues.  It  is  common  for  the 
second  source  to  encounter  production  delays.  Since  the  second  source  is 
usually  initially  assigned  only  small  quantities,  the  total  effect  on  system 
effectiveness  may  be  slight.  Review  of  Lovett  and  Norton  [10]  leads  to  the 
thought  that  another,  albeit  small,  risk  also  exists  that  this  study's 
limited  data  base  also  found.  This  risk  is  the  risk  of  never  getting 
deliveries  from  the  second  production  source. 

D.  DECISION  PROBLEM  IMPACT. 

It  seems  conceivable  that  the  developer's  reliability  is  enhanced  by 
the  mere  presence  of  ar:  alternate  source.  The  second  source's  reliability, 
if  deficient  earl  *  In  pr.  Auction,  does  seem  to  improve  over  time.  In  any 
Case,  the  relatively  small  observed  differences  in  reliability  are  most 
likely  overwhelmed  by  the  conditions  of  actual  use,  e.g.,  gunner  accuracy. 

The  available  data  are  too  sparse  and  the  effect  typically  too  small  to 
make  a  complex  approach  to  dealing  with  schedule  and  reliability  worth¬ 
while.  It  is  recommended  that  the  analyst  combine  judicious  use  of  the 
data  reported  with  system-specific  considerations  to  directly  create  an 
outcome  probability  density  function.  The  ultimate  goal  is  the  same  as  for 
the  cost  objective— to  project  a  measure  of  the  difference  in  outcome  to 
be  expected  if  one  follows  a  competitive  or  sole  source  stragegy.  The 
decision-maker  will  typically  be  able  to  evaluate  directly  the  relative 
importance  of  schedule  and  reliability  differences  among  strategies.  If 
this  is  too  difficult,  then  the  more  elaborate  techniques  of  multi¬ 
attribute  utility  theory  can  be  used  [6]. 
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CHAPTER  III 
DECISION  APPROACH 


A.  INTRODUCTION. 

This  chapter  offers  a  structure  that  provides  a  realistic  approach  to 
the  complex  strategy  selection  problem  with  its  dynamic,  probabilistic, 
multi -objective  nature.  A  first  step  in  problem  analysis  is  to  dissect  the 
problem  in  order  to  clearly  identify  the  key  pieces  that  materially  affect 
the  outcome.  It  is  important  to  consider  these  key  structural  pieces 
whether  or  not  their  values  can  be  accurately  determined  in  a  given  setting. 
The  problem  is  then  structured  to  allow  exploring  the  effects  of  the  key 
factors  on  alternative  strategies.  An  example  of  a  key  factor  is  the 
competitive  environment  present  for  the  buy.  The  state  of  knowledge  of  the 
values  of  these  factors  is  limited.  Not  enough  cases  of  any  given  type 
exist  to  provide  firm  guidance  concerning  outcomes.  The  infeasibility  of 
conducting  experiments  precludes  gathering  information  by  that  means. 

Thus,  the  best  that  can  be  attained  is  to  use  an  approach  employing  all 
available  information  in  the  best  way  possible.  The  mathematical  approach 
explained  below  provides  an  integrating  structure  in  which  one  can  explore 
the  relative  effects  of  the  components. 

Consider  the  structure  of  the  problem.  A  given  competitive  strategy 
and  its  basic  means  of  expression  can  be  illustrated  by  the  example  in 
Figure  6.  The  uncertainty  present  in  tracing  the  series  of  outcomes  over 
the  production  phase  suggests  the  suitability  of  a  stochastic  model.  The 
heart  of  the  cost  model  is  a  stochastic  structure  capable  of  evaluating 
specific  strategies  relative  to  a  baseline  alternative.  The  other  key  part 
of  the  model  is  the  set  of  input  probability  density  functions  that  can  be 
modified  to  meet  the  specific  circumstances  of  a  given  case. 
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Establish  Award  1st  Award  2nd  Award 

Second  Competitive  Competitive  Competitive 


Figure  6.  Example  Illustrating  Competitive  Strategy. 

An  overview  of  the  entire  decision  approach  is  given  in  Figure  7.  In 
applying  the  approach  the  first  step  is  to  apply  a  decision  net  which  will 
reduce  the  number  of  candidate  strategies  through  the  systematic  application 
of  a  series  of  questions.  Then  viable  production  schedules  must  be  prepared 
for  the  surviving  strategies.  Each  suitably  defined  production  schedule  is 
a  required  input  to  the  cost  model.  Several  probabilistic  parameters  must 
also  be  provided,  and  the  entire  acquisition  network  developed  for  each 
candidate  strategy.  One  tool  for  analyzing  such  networks  is  VERT  [13].  The 
novel  use  of  VERT  proposed  here  is  illustrated  in  Section  D  for  a  sample 
problem.  Finally,  the  analyst  must  consider  the  other  objectives  of  the 
decision  in  providing  the  full  array  of  projected  outcomes  to  the  decision¬ 
maker. 

B.  DECISION  SCREEN. 

A  decision  screen  is  a  series  of  questions  used  as  a  device  to 
eliminate  unsuitable  strategies  from  the  analysis  effort.  The  screens 
designated  for  the  approach  are  illustrated  in  Figures  8  and  9.  The  first 
screen  is  concerned  with  program  conditions  while  the  second  screen  focuses 
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Strategic  Alternatives 


♦  ♦ 


Figure  7.  Approach  Schematic 
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Figure  8.  Screen  1  -  Program  Conditions. 


Are  we  early  enough  in  the 
system  process  to  consider  a 
contractor  teaming  develop¬ 
ment  strategy?  Are 
adequate  funds 
available  for  it,  and 
is  it  an  acceptable 
strategy? 


Reject 
contractor 
teami ng 
strategies 


Can  another  legal  strategy 
be  formulated  that  might 
be  applicable  in  this 
situation  (e.g., 
some  type  of 
restricted 
competition? 


Execute  Screen  2 
for  each  general 
strategy  still 
viable 


Figure  8  (Cont’d) 


Is  this  strategy  generally 
acceptable  considering  any 
other  system  specific 
conditions? 


Can  you  lay  out  for  the  candidate 
fundamental  strategy  a  time  phased 
production  schedule  for  which 
funding  is  feasible  and 
operational  requirements  are  Y/ 
met? 


Is  the  strategy  compatible  with 
critical  national  goals 
(e.g.,  international  agree¬ 
ments,  mobilization  base 
needs,  etc.)? 


Reject 


Reject 


Reject 


Carry  the  strategy  forward 
for  detailed  evaluation. 


Figure  9.  Screen  2  -  Strategic  Issues. 
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on  issues  related  to  the  strategy  being  evaluated.  The  screen  can  De 
applied  to  each  potential  strategy  in  turn,  following  the  layout  of  the 
figures.  After  the  viable  general  strategies  are  determined,  the  detailed 
construction  of  schedules  and  strategy  choices  for  each  time  period  can  be 
performed.  These  steps  are  preliminary  to  actually  developing  the  VERT 
network  and  costing  each  strategy  in  a  probabilistic  manner. 

C .  COST  MODEL. 

1 .  Overview . 

Earlier  cost  models  of  the  competitive  versus  sole  source  strategy 
problem  have  been  too  simplistic.  In  [5]  and  [10]  regression  models  were 
proposed  for  use  in  predicting  tne  cost  of  acquiring  systems  under  competi¬ 
tion.  These  models  are  unsatisfactory  because  they  do  not  permit  a 
reasonably  accurate  estimate  of  recurring  cost  savings  and  do  not  recognize 
the  wide  variation  in  possible  outcomes.  The  dependent  variable  in  these 
regressions  is  inappropriate;  typically  it  is  the  projected  competitive 
price.  The  studies  then  show  a  recession  curve  using  this  dependent 
variable  which  nas  a  deceptively  good  fit  (R?).  Seemingly  accurate 
prediction  of  the  competitive  price  is  not  nearly  the  same  as  predicting 
the  savings,  however.  The  underlying  data  are  simply  too  erratic  to 
permit  accurate  projections  of  savings. 

In  [7]  another  approach  is  presented,  but  the  model  in  [7]  applies 
only  to  the  case  of  sustained  split  awards  among  the  two  sources.  Like  the 
models  above,  it  does  not  recognize  the  wide  variations  in  competitive 
environments.  It  indiscriminately  uses  the  historical  data  base  to  predict 
savings  in  the  split  award  environment.  Thus  tnis  approach  extends  without 
modification  the  data  base  findings  too  far  outside  their  legitimate  domain. 


In  fact,  most  of  the  historical  cases  reflected  buy-out  situations  or  a 
mixture  of  split  award  and  buy-out  results.  The  cost  model  presented  in 
this  chapter  improves  on  the  approach  in  [7]  by  reflecting  the  dynamic 
nature  of  the  competitive  environment. 

2 .  Model  Definition. 

The  starting  point  for  the  cost  model  is  the  basic  cost  expression 
of  unit  learning  curve  theory.  According  to  this  theory,  total  recurring 
cost  C  to  the  government  can  be  represented  as  the  sum  of  a  declining  set 
of  unit  costs  represented  usually  by  a  power  function 


where  a  is  the  theoretical  first  unit  cost,  b  reflects  the  rate  of  cost 
improvement,  and  production  extends  from  unit  q-|  to  q2- 

Several  modifications  to  the  basic  expression  in  (1)  are  required 
in  order  to  analyze  the  strategy  problem.  Consider  a  single  delivery  year 
which  is  embedded  in  an  overall  production  effort.  Since  the  particular 
delivery  year  deals  with  only  a  portion  of  the  total  buy,  an  appropriate 
discount  rate  should  be  applied  to  the  recurring  cost  expression.  This 
leads  to  year  i  cost  given  by 

qi  h 

q  =  £  aqD  (2) 

q=qi-l  +  1 

where  q^  represents  the  production  experience  of  a  producer  at  the  end  of 
year  i . 

In  addition  to  several  years  of  production,  several  different 
production  contracts  may  also  be  involved.  Alterations  to  the  competitive 
environment  may  lead  to  expected  cost  effects  reflected  by  the  random 
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variables  1^  and  b,  where  typically  the  expected  value  E(I-j)  <  1  for  more 
competitive  environments.  Then  the  contract  cost  is  given  by 

Qi  (b-Tj) 

Ki  =  Tj  E  aq  (3) 

q=q-j-i  +  i 

V 

where  I.j  measures  a  competitive  shift  in  the  improvement  curve  and  measures 
a  change  in  the  rate  of  improvement.  In  this  particular  expression,  the 
previously  mentioned  discounting  has  been  suppressed  for  purposes  of 
simplicity.  The  values  of  1^  and  b^  and  the  degree  of  production 
experience  are  affected  by  the  previous  production  awards  for  the 

'"V 

system.  The  value  of  I-j  can  also  incorporate  production  rate  shifts. 

These  complexities  suggest  that  a  strategy's  cost  can  best  be 
modeled  as  a  stochastic  network  like  the  example  in  Figure  10.  The  various 
branches  represent  potential  outcomes  of  contract  awards.  The  branches  are 
labeled  with  estimates  of  the  conditional  probability  of  that  particular 
outcome  being  generated.  The  competitive  environment  is  path  dependent  and 
described  by  subjectively  estimated  probability  density  functions.  Some 
examples  are  provided  in  Figure  11.  The  total  set  of  probabilities  to  the 
cost  model  includes  the  non-recurring  costs,  the  recurring  savings,  and 
the  contractor  win  probabilities.  The  other  major  input  is  the  detailed 
strategic  schedule  as  illustrated  by  the  example  in  Figure  12.  For  the 
more  complex  strategies  the  VERT  technique  can  be  used  as  illustrated  in 
section  4.  To  acquire  in-depth  familiarity  with  VERT  the  reader  should 
see  [13j  and  then  acquire  a  VERT  guide  from  the  US  Army  Logistics  Manage¬ 
ment  Center,  Fort  Lee,  VA. 
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Figure  10.  Network  for  Example  Competitive  Strategy 
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gure  11.  Some  Example  Probability  Density  Functions  for 
Percentage  Recurring  Cost  Savings  Relative  to 
Sole  Source. 
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QUANTITY  BY  YEAR 


1  2  3  4  5  6  7 


Developer 

50 

60 

Second  Source 

40 

Split  Award 

250 

500 

(Competitive) 

(60%-40% 

(60%-40% 

MAX.  SPLIT) 

MAX.  SPLIT) 

Competitive  Buy-Out 

500  500  500 

MULTI-YEAR 


Figure  12.  Example  Production  Schedule  for  a  Strategy  Alternative. 
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3.  Remarks. 

The  approach  advocated  here  tries  to  deal  realistically  with  the 
important  parameters  of  the  problem.  It  is  important  to  have  a  certain 
structural  integrity  to  the  model  even  though  it  means  admitting  that  some 
of  the  input  data  in  an  application  is  heavily  judgmental.  If  factors  do 
importantly  affect  the  outcome,  it  is  preferable  to  make  conscious  efforts 
to  evaluate  their  effects  than  to  use  a  more  simplistic  model  in  which  the 
influence  of  these  important  factors  is  suppressed. 

Comments  need  to  be  made  about  certain  specific  characteristics 
of  the  cost  model.  For  example,  the  allocation  of  production  quantity  in 
split-buy  situations  affects  future  cost  projections  because  it  affects  the 
production  experience  of  the  firms.  Particular  assessments  of  outcomes  and 
their  probabilities  can  be  made  judgmentally  for  each  case.  The  allocation 
history  should  not  be  ignored,  however,  for  a  contractor  who  gets  little 
production  experience  will  not  likely  remain  a  viable  competitor. 

Each  individual  case  will  require  the  direct  estimation  of  uncertain 
outcomes  in  a  setting  that  differs  from  all  prior  data  base  cases.  There 
are  no  mathematical  results  such  as  Bayes  Theorem  that  can  reveal  what  the 
probability  distributions  ought  to  be.  It  is  necessarily  a  subjective 
process.  Yet  the  magnitude  of  the  cost  makes  a  formal,  explicit  judgment 
preferable.  For  example,  the  estimated  recurring  cost  savings  is  affected 
by  variations  in  the  competitive  environment,  by  special  contracting 
factors,  and  by  system-specific  production  characteristics.  The  latter 
factor  is  very  important;  it  includes  features  such  as  degree  of  produc¬ 
tion  automation,  percentage  of  total  cost  represented  by  materials,  and 
many  others. 
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In  some  situations  it  will  be  necessary  to  model  various  dependen¬ 
cies.  For  example,  one  of  the  potential  second  sources  may  already  be  a 
system  subcontractor  and  have  available  some  of  the  necessary  production 
equipment.  This  particular  case  may  not  be  common,  but  it  has  occurred 
and  every  real  case  is  likely  to  have  some  peculiarity.  Yet  the  modeling 
approach  recommended  here  is  general  and  ought  to  be  able  to  accommodate 
most  of  these  unusual  features. 

Another  reason  to  avoid  over  ’eneral ization  blindly  from  the  data 
base  is  that  the  cases  studied  have  had  a  conglomeration  of  strategies,  but 
the  results  have  all  been  lumped  together.  That  is,  savings  from  buy-outs 
have  not  been  carefully  distinguished  from  savings  from  split  awards.  It 
should  be  noted  that  different  researchers  tend  to  get  quite  different 
savings  projections  on  the  same  systems  [7,  10].  Also  each  successive 
study  tends  to  be  further  removed  from  the  primary  data.  Finally,  it  can 
be  deduced  from  [7]  that  behavior  may  be  different  for  different  classes  of 
systems.  For  example,  the  missile  system  findings  in  [7]  are  quite 
different  than  the  data  base  average.  What  then  does  one  do  about  systems 
for  which  no  past  data  base  exists  (e.g.,  armor)? 

4.  Example  VERT  Application. 

Suppose  that  the  Little  Boy  missile  system  is  nearing  the  stage  of 
development  where  a  serious  analysis  of  its  planned  production  strategy 
needs  to  be  conducted.  Assume  that  the  decision  screen  described  above 
has  been  applied  and  that  two  strategies--one  sole  source  and  one 
competitive --need  detailed  outcome  projections. 

The  competitive  strategy  consists  of  the  following  stages: 

a.  Selection  of  a  second  source  and  award  of  a  small  initial  buy 
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b.  Award  of  a  split  buy  to  the  two  sources  with  the  award 
percentages  determined  by  the  relative  bid  prices 

c.  Award  of  a  second  split  buy  with  the  same  allocation  technique 
as  used  previously,  but  with  larger  production  quantities 

d.  A  multi-year  buy-out  with  the  total  award  quantity  going  to  the 
lowest  offeror. 

The  schedule  and  contracting  plan  for  this  strategy  was  given  in  Figure  12. 

A  schematic  view  of  the  strategy  dynamics  was  given  in  Figure  10.  The  arcs 
emanating  from  the  split  buy  points  show  the  different  allocation  decisions 
possible  for  this  case.  The  outcomes  in  the  last  block  can  be  measured 
with  reference  to  a  baseline  sole  source  strategy.  Note  that  the  result 
expected  at  the  buy-out  point  depends  on  the  outcome  of  the  previous  split- 
award  competitions.  For  example,  if  the  second  source  has  not  shown  itself 
as  a  viable  cost  competitor  during  the  split  awards,  very  little  competitive 
pressure  may  be  present  during  the  buy-out  situation. 

key  "most  likely"  estimates  for  the  baseline  sole  source  strategy 
are  as  follows: 

Theoretical  first  unit  cost:  a  *  $10,000. 

Cost  improvement  curve  slope  of  90%.  This  translates  into 
b  =  -.152  in  the  learning  curve  formula:  Unit  cost  =  a  xb. 

In  order  to  compare  the  cost  results  of  the  competitive  strategy 
with  those  likely  under  the  sole  source  alternative,  the  estimation  of 
several  input  parameters  is  required.  The  values  of  these  inputs  are  not 
accurately  established  by  available  data.  Yet  nothing  is  gained  by 
neglecting  them,  for  they  do  impact  the  results.  The  best  approach  is  to 
extract  the  maximum  information  regarding  these  parameters  using  both  the 
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limited  data  base  and  informed  judgment.  By  this  means  the  required 
probability  density  functions  can  be  selected.  Some  examples  were  described 
in  Figure  11.  Note  that  some  of  these  functions  depend  on  the  outcomes  of 
preceding  buys.  Contractor  win  probabilities  have  also  been  estimated,  and 
the  entire  structure  was  summarized  pictorially  in  Figure  10. 

VERT  simulation  of  the  complete  example  case  described  above  was 
performed  for  500  iterations  with  the  results  shown  in  Figure  13.  The 
required  VERT  input  necessary  to  describe  the  network  is  listed  in 
Appendix  C.  This  application  of  VERT  is  unusual  because  key  data  reflect¬ 
ing  experience  and  also  the  competitive  density  functions  were  trans¬ 
mitted  through  parts  of  the  VERT  program  designed  to  carry  time  and 
performance  information.  In  the  example  presented  here  the  baseline  sole 
source  strategy  has  a  cost  of  $3,037,000  which  can  be  easily  computed  from 
the  learning  curve  assumptions.  In  this  example  the  two  strategies  have 
the  same  annual  production  schedules,  and  production  rate  adjustments  (such 
as  may  occur  for  split-buys)  do  not  affect  the  unit  costs.  The  model  can 
be  altered  to  handle  variations  of  these  conditions,  however.  A  non-zero 
discount  rate  can  also  be  applied.  In  the  present  case  Figure  13  shows 
that  only  about  20%  of  the  time  would  the  competitive  strategy  be  lower- 
priced  than  the  sole  source  alternative.  The  cost  outcome  would  be  coupled 
with  the  other  decision  factors  discussed  in  this  report  in  order  to  arrive 
at  a  decision. 
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OVERALL  COST  FOR  NODE  CAD 
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MULTIMODAL  DISTRIBUTION 

Figure  13.  Total  Cost  Distribution  for  a  Competitive  Strategy. 
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CHAPTER  IV 

DISCUSSION  AND  SUMMARY 

The  studies  analyzing  the  cost  behavior  of  systems  acquired  competi¬ 
tively  have  reported  varying  levels  of  competitive  savings  due  to  lower 
unit  recurring  costs.  Despite  the  wide  variation  in  the  reported  savings, 
the  studies  consistently  agree  that  the  potential  savings  are  substantial. 
This  study  reported  the  limited  state  of  the  art  in  modeling  problems  of 
acquisition  strategy  choice.  Key  conditions  which  affect  the  outcomes 
likely  for  a  given  system  were  described.  Progress  in  dealing  with  such 
related  problems  as  the  effect  of  production  rate  on  system  cost  was  also 
described. 

This  study  marked  the  first  empirical  attention  to  non-cost  issues  in 
strategy  selection.  In  particular,  findings  concerning  the  reliability  and 
schedule  effects  of  second  sources  were  reported.  A  tentative  conclusion 
was  that  the  reliability  objective  should  be  given  little  weight  in  the 
strategy  selection  decision  provided  adequate  government  acceptance  testing 
is  planned.  Initial  schedule  delays  involving  second  sources  are  common. 
The  importance  of  this  consideration  depends  on  the  volume  of  production 
affected  and  on  the  significance  of  a  schedule  delay  for  the  given  system. 

One  of  the  aims  of  this  study  was  to  provide  an  approach  to  analyzing 
the  acquisition  strategy  problem.  To  accomplish  this  purpose,  a  decision 
screen  and  a  cost  model  were  presented.  The  overall  approach  involved 
applying  a  screen  in  order  to  focus  attention  on  the  key  strategies. 
Feasible  production  schedules  are  required  for  the  remaining  strategies 
in  order  to  perform  detailed  cost  evaluation.  Finally,  assessment  of 
effects  on  other  objectives  is  also  required  prior  to  choice  of  the  best 
strategy. 


33 


A  highlight  of  the  cost  model  is  the  attempt  to  closely  integrate  and 
provide  for  contributions  from  each  of  the  areas  of  methodology,  historical 
data,  and  judgment.  The  paper  contends  that  such  a  blending  is  demanded 
for  an  approach  that  realistically  focuses  on  the  key  issues  in  selecting 
an  acquisition  strategy.  The  structure  of  the  problem  and  the  limited 
number  and  type  of  cases  comprising  the  data  base  lead  to  the  importance  of 
a  systematic  application  of  judgment. 

For  relatively  complex  strategies,  it  was  shown  that  a  stochastic 
network  could  represent  the  dynamics  of  the  acquisition  process.  The 
applicability  of  combining  parametric  cost  estimates  with  the  Venture 
Evaluation  and  Review  Technique  (VERT)  to  analyze  cost  outcomes  was  also 
i 1 1 ustrated . 

Several  future  research  steps  are  possible.  Advances  specific  to  the 
approach  recommended  here  include  refining  of  the  decision  screen  as  it  is 
used  by  actual  project  management  in  a  variety  of  circumstanced.  Automated 
computing  refinements  could  also  make  the  linkage  between  cost  estimating 
tools  and  the  VERT  network  automatic.  More  generally,  a  great  need  exists 
to  investigate  fully  cost-saving  behavior  under  multi-year  contracts  [3]. 
The  ideas  discussed  in  this  report  may  also  be  applicable  to  strategy 
selection  for  phases  prior  to  production. 
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APPENDIX  A 


SECOND  SOURCE  RELIABILITY  FINDINGS 

Findings  relating  to  the  reliability  of  second  source  production  are 
described  in  this  appendix.  All  systems  for  which  such  data  were  located 
are  reported.  The  completeness  of  the  available  information  varied  from 
informal  generalities  in  personal  interviews  to  extensive  test  results  for 
thousands  of  missile  firings.  The  following  table  describes  the  type  of 
information  and  the  principal  findings  for  the  systems.  Following  the 
table,  an  analysis  of  two  Navy  missile  system  cases  is  included.  This 
analysis  was  provided  to  the  researchers  by  personnel  of  the  Pacific  Missile 
Test  Center,  Point  Mugu,  California. 
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A  COMPARISON  OF  MISSILE  QUALITY 
FROM 

PRIMARY  AND  SECOND  SOURCE  CONTRACTORS 


SUMMARY 

Data  packages  for  two  air-launched  missiles  were  developed  by  the  Naval 
Weapons  Center  (NAVWPNCEN),  China  Lake.  For  each  data  package  two  contractors 
(  a  primary  and  a  second  source)  were  selected.  Extensive  monitoring  was 
performed  to  assure  that  the  contractors  could  produce  in  accordance  with  the 
data  packages. 

The  quality  of  the  production  missiles  was  verified  by  tests  at  the 
Pacific  Missile  Test  Center  (PACMISTESTCEN) ,  Point  Mugu.  Corrective  actions 
were  taken  when  warranted. 

It  is  concluded  that  second  source  contractors  can  produce  missiles  of 
equivalent  quality  to  primary  contractors.  It  would  appear  that  production 
quality  cannot  be  assured  by  the  data  package.  Fabrication  techniques,  workman¬ 
ship,  and  quality  assurance  techniques  unique  to  the  contractor  can  significantly 
influence  production  quality.  The  timely  detection  of  quality  problems  is 
aided  by  a  production  verification  test  program. 
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A  COMPARISON  OF  MISSILE  QUALITY 
FROM 

PRIMARY  AND  SECOND  SOURCE  CONTRACTORS 


INTRODUCTION 

The  Production  Acceptance  Test  and  Evaluation  Division  of  the  PACMISTESTCEN 
is  conducting  production  verification  test  and  Government  lot  acceptance  test 
programs  with  the  objective  of  verifying  the  performance  and  reliability  of 
production  missiles. 

This  report  presents  a  comparison  of  the  test  data  on  missiles  from  two 
procurement  sources  and  the  general  deductions  which  can  be  reached.  In  view 
of  the  nature  of  the  test  programs,  which  were  designed  to  accept  production 
missiles  on  the  basis  of  tests  conducted  on  a  sample,  the  data  is  not  well 
suited  to  support  strong  conclusions  about  differences  which  might  exist 
between  the  quality  of  missiles  from  two  sources. 

The  data  packages  were  developed  by  NAVWPNCEN,  China  Lake.  Very  extensive 
monitoring,  including  tests  at  the  PACMISTESTCEN,  was  performed  to  assure  that 
the  contractors  could  produce  in  accordance  with  the  data  packages.  During 
the  production  years  monitoring  was  continued  at  the  level  necessary  to  assure 
that  performance  and  reliability  were  maintained. 

RESULTS  AND  DISCUSSION 

Missile  A 


Missile  A  is  an  air-to-air  guided  missile.  A  sample  of  15  guidance 
sets  was  selected  from  each  contract  year  production  and  subjected  to  individual 
tests  and  environmental  failure  rate  tests. 

The  individual  tests  are  functional  tests  conducted  with  the  use  of  the 
approved  guided  missile  test  set.  The  desired  yield  of  the  test  is  95  percent. 

The  environmental  failure  rate  test  (reliability  test),  is  a  laboratory 
simulation  of  the  captive-flight  parameters  that  contribute  to  random  failures. 
The  desired  mean-time-to- failure  (MTBF)  is  450  hours. 

Table  1  shows  a  summary  of  data  from  the  conducted  tests. 

The  second  source  contractor  delivered  pilot  production  guidance  sets  for 
contract  year  1975.  When  a  sample  of  these  guidance  sets  was  subjected  to  the 
reliability  test  it  demonstrated  an  MTBF  of  30  hours.  The  problems 
were  related  to  substandard  vendor  components  and  production  processes.  By 
taking  33  corrective  actions  and  conducting  further  reliability  tests  in  the  test 
analyze  and  fix  mode,  the  MTBF  was  increased  to  300  hours. 
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Guidance  sets  from  the  second  source  contractor  for  contract  years  1976 
and  1978(1)  demonstrated  early  failures  when  subjected  to  the  reliability  test. 
The  problems  were  not  detected  until  the  independent  production  verification 
tests  were  conducted.  The  problems  were  attributed  to  substandard  components, 
workmanship,  and  quality  assurance  techniques.  The  guidance  sets  were  pro¬ 
duced  in  accordance  with  the  data  package. 

The  guidance  sets  for  contract  year  1976  were  combined  with  those 
for  contract  year  1975  in  a  retrofit  program.  The  guidance  sets  for  contract 
year  1978(1)  were  subjected  to  an  extensive  inspection  and  screening  program 
and  a  second  sample  designated  as  contract  year  1978(2)  was  selected. 

The  data  show  that  guidance  sets  produced  by  the  primary  contractor 
meet  the  individual  test  and  reliability  test  desired  yield  of  95  percent  and 
desired  MTBF  of  450  hours.  The  second  source  contractor  has  experienced 
difficulty  in  producing  guidance  sets  which  meet  the  test  objectives;  however, 
the  problem  appears  to  be  fabrication  techniques,  workmanship,  and  production 
processes  unique  to  the  contractor. 

Missile  B 


Missile  B  is  also  an  air-to-air  guided  missile.  A  sample  of  15 
guidance  control  sections  was  selected  from  each  lot  within  the  contract  year 
production  and  subjected  to  depot  tests  and  flight  test  simulatioo. 

The  depot  tests  are  functional  tests  conducted  in  accordance  with 
Naval  Air  Systems  Command  specifications  using  automatic  test  equipment.  The 
desired  yield  of  the  depot  test  is  95  percent. 

The  flight  test  simulation  (reliability  test)  uses  a  composite 
mission  profile  derived  from  response  measurements  aboard  the  carrying 
aircraft.  The  simulation  reproduces  in  level,  sequence,  and  combination 
the  stresses  of  temperature,  operation,  shock,  and  vibration  that  are  the 
primary  contributing  factors  to  failure.  The  desired  MTBF  is  450  hours. 

Table  2  shows  a  summary  of  data  from  the  conducted  tests. 

It  can  be  seen  that  both  contractors  produce  guidance  control 
sections  which  meet  the  depot  test  and  reliability  test  objectives  of 
95  percent  and  450  hours.  In  addition,  both  contractors  not  only  exceed 
the  reliability  test  objective,  but  exhibit  a  trend  for  an  increasing  MTBF. 

CONCLUSIONS 

Second  source  contractors  can  produce  missiles  of  equivalent  quality  to 
primary  contractors. 

Production  quality  cannot  be  assured  by  the  data  package. 

Production  quality  can  be  significantly  influenced  by  fabrication 
techniques,  workmanship,  and  quality  assurance  techniques. 

Timely  detection  of  quality  problems  is  aided  by  a  production  verification 
test  program. 
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Table  1.  Missile  A,  Guidance  Set  Test  Results 


Primary  Contractor 


Contract 

Individual 

Test 

Rel i abi 1 i  ty 

Test 

Year 

Tested, 

Failed 

,  Yield 

Hours, 

Failed 

,  MTBF 

1975 

15 

0 

100.0% 

720 

1 

720 

1976 

15 

0 

100.0% 

1390 

4 

348 

1977 

15 

0 

100.0% 

459 

0 

— 

1978 

15 

2 

86.7% 

958 

2 

479 

1979 

15 

1 

93.3% 

950 

1 

950 

1980 

15 

1 

93.3% 

945 

2 

473 

Overall 

90 

4 

95.6% 

5422 

10 

542 

Second  Source 

Contractor 

Contract 

Individual 

Test 

Rel  i  abi  1  i  ty 

Test 

Year 

Tested, 

Failed 

,  Yield 

Hours, 

Failed 

,  MTBF  . . 

1976 

21 

1 

95.2% 

1379 

7 

197 

1977 

15 

0 

100.0% 

Assets 

used  for  Flight 

Test  Program 

1978(1) 

15 

3 

80.0% 

1319 

4 

330 

Subtotal 

51 

4 

92.2% 

2698 

11 

245 

1978(2) 

15 

100.0% 

950 

2 

475 

1979 

15 

2 

86.7% 

714 

1 

714 

1980 

15 

100.0% 

Test 

in  process 

Subtotal 

45 

2 

95.6% 

3 

553 

*Table  separated  to  distinguish  initial  problems  and  to  illustrate 
compliance  in  following  years. 
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Table  2.  Missile  8,  Guidance  Control  Section  Test  Results 


Primary  Contractor 


Contract  Depot  Test  Reliability  Test 

Year _ Tested,  Failed,  Yield _ Hours,  Failed,  MTBF 


1977 

48 

3 

93.7% 

6206 

8 

776 

1978 

120 

5 

95.8% 

9355 

5 

1871 

1979 

60 

1 

98.3% 

5170 

2 

2585 

Overall 

228 

9 

96.1% 

20,731 

15 

1382 

Second  Source  Contractor 


Contract 

Year 

Depot  Test 

Tested,  Failed,  Yield 

Reliability  Test 

Hours,  Failed,  MTBF 

1977 

72 

3 

95.8% 

8733 

9 

970 

1978 

48 

2 

95.8% 

4490 

1 

4490 

1979 

60 

3 

95.0% 

5356 

3 

1785 

Overall 

180 

8 

95.6% 

18,579 

13 

1429 
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APPENDIX  B 


DELIVERY  SCHEDULE  FINDINGS 


The  first  part  of  this  appendix  consists  of  twelve  graphs  contrasting 
the  schedule  performance  of  original  and  second  source  producers  of  six 
weapon  systems.  The  first  six  graphs  compare  the  two  sources  at  equivalent 
production  quantities.  The  final  six  graphs  compare  the  sources  at  equiva¬ 
lent  times  past  their  original  initial  delivery  date.  In  all  cases  the 
vertical  axis  measures  the  months  from  scheduled  delivery  to  actual  delivery 
of  the  required  quantity.  A  positive  value  is  measuring  a  late  delivery, 
and  a  negative  value  reflects  ahead  of  schedule  performance. 

Following  the  graphs,  a  table  reports  additional  findings.  For  most  of 
these  cases,  the  available  information  was  insufficient  to  create  a  graph. 


All  identified  cases  are  reported  either  in  the  graphs  or  in  the  table. 


i 


DELIVERY  PERFORMANCE  VERSUS  SCHEDULED  PRODUCTION 
NAVY  KESSJLE  SUBSYSTEM  B 
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ADDITIONAL  SCHEDULE  INFORMAT I ON 


SYSTEM 


COMMENTS 


1.  Army  laser  range  finder 

(This  case  was  also  graphed.) 


2.  Missile  subsystem 

3.  Army  missile 

4.  Army  missile  subsystem 

5.  Army  missile 

6.  Army  missile  subsystem 


7.  Army  night  vision  system 

(This  case  was  also  graphed.) 


8.  Army  ground  radar  system 


Government  sources  reported  that  an 
inadequate  TDP  delayed  the  second 
source  contractor.  The  second 
source  was  used  before  the  TDP  was 
under  Government  configuration 
control.  In  the  second  source's 
early  phase  of  production  2000 
changes  were  issued  incrementally. 
After  the  initial  period,  the  second 
source's  performance  was  suitable. 

Second  source  had  only  delivered  for 
about  three  months  but  was  on 
schedule. 

Second  source  had  some  delays  due  to 
GFE  but  otherwise  met  schedule. 

Prime  only  missed  schedule  a  few 
times  which  was  also  due  to  GFE. 

Second  source  had  difficulty  first 
few  months,  then  met  schedule. 

Second  source  delivered  original 
contract  within  a  few  weeks  of 
schedule. 

Government  sources  said  a  poor  TDP 
caused  second  source  problems  during 
the  educational  buy.  There  were  many 
changes.  This  led  to  a  decision  to 
have  a  split  award  following  the 
educational  buy. 

Government  sources  said  a  TDP 
problem  (lack  of  change  control)  was 
the  biggest  problem.  The  original 
source  was  obstinate  and  refused  to 
talk  with  the  second  source. 

Second  source  was  a  technically 
unqualified  small  business  and  never 
delivered  a  complete  system. 
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EXAMPLE  VERT  INPUT 


NQN_R£C  BEGIN  SAN1  1.0 


Na.N.RECOCQST  1  '  3.0  1000000. 

2000000. 

1500000. 

SAAi;  5  AN  1  SAM  12  1.0 

S  AA  1 IGPERf  1  3.0 

4.0 

10.0 

7-0 

SxAURCuSX  1340K  l.OP 

SAA 11 M  1  .2 

SAM1K 

100. 

IS 

1 

SAA12  SAN1  SAN122  1.0 

S  AA 120PE RF  1  3.0 

4.0 

10.0 

7.0 

SAA12RCUSI  1340K  l.OP 

SAA 12N  1  .3 

SAA12X 

100. 

IS 

1 

SAA  13  SAM  SAM  32  1.0 

SAA130PERF  1  3.0 

4.0 

10.0 

7.0 

SAA13RC0ST  1 340K  l.OP 

SAA13K  1  .5 

SAA13K 

100. 

IS 

1 

SAA21  S  AN  1 1 2  SAN21  1.0 

SAA210PERF  1  3.0 

7.0 

14.0 

10.0 

SAA21RCQST  134QK  l.OP 

SAA21K 

too. 

IS 

1 

SAA21M  1  • b 

S  A A  22  SAN112  SAN22  1.0 

SAA220P ERF  1  3.0 

7.0 

14.0 

o 

• 

O 

H 

SAA22RCQST  1 340X  l.OP" 

SAA22H  1  .2 

SAA22K 

100. 

IS 

l 

SAA23  SAN] 12  SAN23  1.0 

SAA23DPERF  1  3.0 

7.0 

14.0 

10.0 

SAA23RC0  ST  1340K  l.OP 

3AA23H  1  .2 

SAA23K 

100. 

IS 

1 

SAA24  SAN122  SAN24  1.0 

5 AA24UPERF  1  3.0 

$.0 

16.0 

12.0 

3AA24RC0S7  1 34  OK  l.OP 

SAA24P.  1  .3 

SAA24K 

100. 

IS 

1 

SAA25  S  AN  12 2  SAN25  1.0 

3AA253PERF  1  3.0 

9.0 

16.0 

12.0 

SAA2SRCGST  1340K  1.0? 

SAA25M  1  .45 

SAA25K 

100. 

IS 

1 

SAA26  S  AN  122  SAN26  1.0 

3AA260PEHF  1  3.0 

9.0 

16.0 

12.0 

SAA26RC0ST  1340K  l.OP 

SAA26H  1  .25 

SAA26K 

100. 

IS 

1 

SAA27  SAN132  SAN27  1.0 

SAA270PERF  1  3.0 

9.0 

16.0 

12.0 

SAA27RC0  ST  1340K  l.OP 

SAA27K 

100. 

IS 

1 

SAA27H  1  .25 


•  OK  1147 329. K 


•  OK  1147 329  . K 


•0K1147329.K 


.  0K2029  385 . K 


.  OK2029365.K 


.OK2029385.K 


. 0K2051180.K 


-0K2051180.K 


. OK  205 1 1 80  •  K 


. 0K2029365.K 
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APPENDIX  C  (Cont'd) 


$  A  A  i:  3  B  AN l 3  2  SAN28  1.0 


SAA2aop£fir  i 
oA A233CQST  1340K 
SAA28M  1 

3.0 

l.OP 

.25 

9.0 

SAA28K 

16.0 

100.  IS 

12.0 

1.0K2029365.K 

1.0 

5AA29  SAN132 
$«A29GPEf <F  1 
SAA293CC  ST  1340K 
SAA29M  1 

SAN29  1.0 
3.0 

l.OP 

.5 

9.0 

SAA29K 

16.0 

100.  IS 

12.0 

1.0K2029385.K 

1.0 

B0A1  SAN21 

80A10PERF  1 
dOA IRCGSI  l 34 OK 

END  1.0 
3.0 
l.OP 

e.o 

BO  AIK 

22.0 

100.  IS 

14.0 

1.0K4778050.K 

1.0 

b  3A2  SAN22 

S0A2GPERF  1 
B0A2RCCST  1340K 

END  1.0 
3.0 
l.OP 

10.0 

80A2K 

23.0 

100.  IS 

ie.o 

1. OK  481 22 16. K 

1.0 

BGA3  SAN23 

BOA3uP£RF  1 
oOA33CCST  1340K 

END  1.0 
3.0 
l.OP 

10.0 

80A3K 

23.0 

100.  IS 

18.0 

1 . 0K4776  050. K 

1.0 

dO  A4  SAN24 

BO A4GP£ RF  1 
80A4RCGST  1340K 

ENO  1.0 
3.0 
l.OP 

10.0 

30A4K 

23.0 

100.  IS 

16.0 

1.0K4794870.K 

1.0 

1 

80A5  SAN25 

BG A50PE RF  1 
dOA5PCGST  134QK 

ENO  1.0 
3.0 
l.OP 

12.0 

30A5K 

30.0 

100. 

IS 

20.0 

1.0K4830130.K 

1.0 

B0A6  SAN26 

83A60PESF  1 
dOASRCOST  1 340K 

ENO  1.0 
3.0 
l.OP 

10.0 

80A6K 

26.0 

100. 

IS 

16.0 

1. OK  47 94  870. K 

1.0 

1 

30A7  SAN27 

33A7GPEPF  1 

30 A7RC0ST  1340K 

ENO  1.0 
3.0 
l.OP 

12.0 

80A7K 

30.0 

100. 

IS 

20.0 

1.0K477  6  050 • K 

1.0 

\ 

dOAd  SA.N28 

BSA3JPERF  1 

30 A3RC0ST  1340K 

ENO  1.0 
3.0 
l.OP 

12.0 

80A6K 

30.0 

100. 

IS 

20.0 

1.0K4612216.K 

1.0 

1 

d2.A9  SAN2  9 

dOA90?ERF  1 
60A9RC0ST  1340K 

ENO  1.0 
3.0 
l.OP 

9.0 

80A9K 

16.0 

100. 

IS 

12.0 

1.0K47780S0.K 

1.0 
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